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ABSTRACT 



This report summarizes a study that analyzed the state 
education accountability reports of 48 states and Washington, DC, and the 
inclusion of students with disabilities in the accountability systems. 

Extreme variability was found to exist in the reporting practices across 
states and there was little information on students with disabilities. Only 
11 states included test-based outcome data for students with disabilities in 
their reports; 5 states produced a separate special education report. Many 
states included input/context and/or process indicators for students with 
disabilities, but few reported outcome data for these students. Finally, 
every state was unclear on at least one educational indicator in their 
accountability system and whether students with disabilities were included. 
The report includes the following recommendations regarding reporting on 
students with disabilities: (1) be clear about the role of students with 

disabilities in data and standards; (2) while special education reports can 
be useful in providing detailed information about students with disabilities, 
general accountability reports should also include information on students 
with disabilities; and (3) when making comparisons among schools, districts, 
and states, comparisons should be based on similar populations. (Contains 28 
references . ) (CR) 
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State Accountability Reports: 

What are States Saying about Students with Disabilities? 



Martha L. Thurlow, Karen Langenfeld, J. Ruth Nelson, Hyeonsook Shin, and Jo Coleman 

National Center on Educational Outcomes 
University of Minnesota 



Abstract 

Public reports on education reflect the ways in which accountability is handled by the state. The 
extent to which students with disabilities are included in these reports is an important index of the 
extent to which responsibility is taken for the education of these students. We analyzed 
accountability reports sent to us by 48 states and Washington, DC. Extreme variability existed in 
reporting practices across states, and there was little information on students with disabilities. 
Only 1 1 states included test-based outcome data for students with disabilities in their reports; 5 
states produced a separate special education report. Many states included input/context and/or 
process indicators for students with disabilities, but few reported outcome data for these students. 
Finally, every state was unclear on at least one educational indicator in their accountability system 
as to whether students with disabilities were included. Several recommendations are made 
regarding clear and inclusive reporting practices. 



During the past five to ten years of education reform, policymakers, school administrators, 
legislators, and the general public have pressed to know whether education is working for 
students. In response to this, there have been developed outcomes-based accountability systems, 
which are systematic methods to assure those inside and outside the educational system that 
schools and students are moving toward desired goals (Brauen, O’ReiUy, & Moore, 1994). An 
accountability system is more than the assessment of outcomes; it requires that student performance 
be reported routinely and that consequences follow (Brauen et al., 1994). According to Brauen et 
al., consequences may be distributed to either individuals, such as a student or teacher, or to 
systems, such as a program or school. Consequences may include sanctions, such as failure to 
graduate or loss of personnel, and/or rewards, such as public recognition of success or increased 
funding (Bond, Braskamp, & Roeber, 1996; Brauen et al., 1994; Geenen, Thurlow, & Ysseldyke, 
1995). 



The most common way of documenting progress for accountability systems has been 
through the development of state accountability reports that regularly present indicators of the 
status of public education, including student assessment data, data on students and teachers, and 
school finance data (CCSSO, 1995). Indicators are statistical measures of the educational system 
(College of Education & Human Development, 1996). In 1995, the Council of Chief State School 
Officers (CCSSO) produced a document on state accountability reports based on a survey of state 
assessment directors. A partnership between CCSSO and the North Central Regional Educational 
Laboratory (NCREL) annually produces the State Smdent Assessment Program Database, which 
includes data from annual surveys of state assessment directors. These reports summarize state 
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reporting purposes, levels of statistics reported, frequency of reporting, whether these reports were 
mandated, and whether rewards and/or sanctions were attached to reporting in the state (Bond, 
Braskamp, & Roeber, 1996; CCSSO, 1995). 

Until recently, little information was available on where students with disabilities fit into 
state accountability systems. Ysseldyke (1995) as quoted by Bond, Braskamp, and Roeber (1996) 
noted that 6 to 14 percent of the total tested elementary population and 5 to 10 percent of the total 
tested high school population were students with disabilities. Forty-one states allow students with 
disabilities to be excluded from the state assessment program (Thurlow, Scott, & Ysseldyke, 
1995). These students can be exempted from the assessment if the assessment is not appropriate 
for them (content is not included in student's lEP), and for most states, a student is included or 
excluded based on lEP recommendations. 

It also has been reported that even when students with disabilities participate in the 
statewide assessment program, their scores might not be included in the state, district, or school 
averages (Erickson, Thurlow, & Ysseldyke, 1996). Thurlow, Scott, and Ysseldyke (1995b) 
further found that of the 24 states that described what they do with data on students with 
disabilities in their written guidelines, more than half (n=14) stated that data from students with 
disabilities were not included in their accountability reports; however, eight states did document the 
number of students excluded from the assessments. According to Elliott et al.(1996), it is not 
uncommon for those students with disabilities who do participate in assessments to have their 
scores deleted, their results shared only with parents, or no record kept of their test. 

States vary in their reporting practices. As Elliott et al. (1996) pointed out, some states 
make decisions about reporting based on whether a student receives an accommodation or on what 
type of accommodation the student receives. It is also not uncommon for students with disabilities 
to be excluded from participation rates for the assessment. Elliott et al. contend that if students 
with disabilities are not assessed in some manner and their results are not reported, then 
accountability for the quality of their educational experience may be compromised. 

To better understand the status of students with disabilities, we examined actual 
accountability reports from states, rather than analyzing policies or personnel reports about 
practice. We looked at the kind of accountability data reported, the presence of comparative data, 
and the breakdown of results for groups of students, particularly students with disabilities. Thus, 
our focus was on information that was not included in the CCSSO survey and State Student 
Assessment Programs Database. Our primary objective was to document the extent to which 
students with disabilities and their assessment results are included in state accountability reports, 
and to recommend ways to move toward systematic and inclusive reporting practices. 
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Method 

The accountability documents for this analysis were obtained by calling the state 
assessment directors or their designees in each of the 50 states and Washington DC, beginning in 
fall 1995. The calls were made based on the information from the CCSSO (1995) report, which 
listed accountabihty documents for each state. Unless otherwise indicated, our analysis is based 
solely on the information contained in the written documents, and not based on other sources of 
information. Accountability reports were collected from all states except California (testing 
program currently under revision), Minnesota (currently developing an accountability system), and 
Wyoming (no formal accountability documents available). We did not include reports of special 
studies or evaluations of specific programs in our analysis. 

Each document was thoroughly examined, using a checklist of commonly used indicators 
of educational performance, and descriptive notes were made regarding each accountability 
document. The indicators were selected by first perusing several existing models of education 
(e.g., Oakes, 1986; Shavelson, McEX>nnell, & Oakes, 1989, Shavelson, McEX>nnell, Oakes, & 
Carey, 1987). In addition, several of the documents we had received were scanned to determine 
whether additional types of indicators were needed. The checklist and descriptive notes were 
completed simultaneously. Raters started at the beginning of each report, marking the indicators 
present and writing notes. Any information found on students with disabilities was noted. 

Limitations and Cautions 



This analysis was limited by some of the following factors: 

- Accountability systems are changing, dynamic processes. A detailed 
analysis of this type will necessarily have some information that is out of date by 
the time of publication. In addition, the documents we obtained contain data 
ranging from the 93-94 school year to the 95-96 school year. 

- Different accountability systems have different definitions of 
accountability. Our requests were answered by state assessment directors in 
different ways. For example, some states included reports based solely on financial 
information. Other states may have had the same types of reports, but did not 
consider them to be accountability documents so they did not send them. 

This analysis is based on documents, not interviews or other sources 
of information. Exceptions include accountability information obtained through 
the World Wide Web. 

This analysis includes over 100 documents, obtained at various times 
from the fall of 1995 through May 1997. Many people have contributed to 
the daunting task of reading, and coding each report. Every effort was made to 
maintain internal consistency and reliability throughout this process. This was 
accomphshed by repeated rechecking of coding, and final consistency checks by 
two individuals. 
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Results 



The following sections contain summary information about the accountability documents 
given to us by each of the states and Washington, DC. The information is presented in greater 
detail in Technical Report 20, and a set of state summaries, which are in a separate 
document available from NCEO (Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1997). 

How are the data reported? 

There is a great amount of variability in accountability reporting across states. There is also 
variability within state’ documents when more than one report is produced by a single state. As 
shown in Table 1, a number of states compile their accountability data into a single report (n=16). 
Thirty-two states provided multiple accountability documents. Accountability documents ranged in 
length from two to approximately 600 pages. Unique documents included small pocket size 
reports or posters with general summary information. 

Level of data . States also varied considerably in the level of data reported, which refers to 
the level of detail given in the accountabihty documents. Reports may provide state, district, 
and/or schoolwide averages for test data during a particular school year. States may have any 
combination of state, district, and school-level reporting, and these may vary from document to 
document, and from one indicator to another. Most states do provide accountability data at the 
state and district levels (N=41). Twenty -eight states actually report data at all three levels. Four 
states provide only state level data (Colorado, Iowa, Montana, and Oregon). 

Focus of reporting . We looked at the focus of reports as being directed toward 
inputs/contexts, processes, or outcome indicators, where indicators are statistical measures of 
some aspect of the educational system (College of Education & Human Development, 1996). Our 
analysis revealed a great deal of variabihty in the focus of reports. In the analysis, 
“inputs/con texts” refers to accountability indicators that describe the student’s learning environment 
as well as demographic characteristics, financial and human resources of the particular district’s 
population (e.g., student-teacher ratio, cost) (1991). Process indicators describe student 
participation and school district evaluation (e.g., enrollment, attendance, accreditation status). 
Outcome indicators are non test and test data indicators that focus on the end result of a student’s 
learning process or are indices of the products of a reciprocal interaction between the individual and 
school or life experiences. 

Of the states that sent us information, all but four (Delaware, Kentucky, Montana, and 
Washington) included information on the inputs/contexts of education for regular education 
students. All states but four (Delaware, Idaho, Montana, Washington) included information on the 
educational process. All states included information on the outcomes of education. 
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Purpose of Reporting. The majority (N=32) of states fell into the “other” category, with 
the majority of these stating the purpose of their reports was “to provide information,” without 
mention of why the data were collected, or how the data should be used. Eight states use the 
information in their accountability reports for accreditation purposes, while five states report 
financial aid as a purpose of the accountability system. Technical assistance was cited as a 
purpose/consequence of the information in four states’ accountabihty reports. Only one state used 
the information reported in its accountabihty document for diploma purposes. Twenty-eight states 
did not mention a purpose in at least one of their accountabihty documents. 

High Stakes Testing Many reports do not mention the stakes attached to particular tests, or 
do not make the stakes of testing clear. For example, Texas only mentions that the TAAS (Texas 
Assessment of Academic Skills) is a graduation exit exam in the glossary section of one of its 
documents. Several documents mention the TAAS without mentioning the stakes. Ohio sent us 
data without mention of how any of the tests reported are used. Of the 17 states that had a 
graduation exam at the time of this report, only one state (Virginia) stated this purpose clearly at the 
very beginning of the document when commenting on the purposes and consequences of its 
accountabihty system. 

Target audience . The majority of states do not specify who their target audience is in the 
introduction to at least one of their reports, or in the title of their reports (N=31). While not shown 
on the figure, we also counted 14 states that did not specify a target audience in any of their 
reports. It is evident that: 

• The most common target audience of reports is the general public (27 states). 

• 18 states direct their reports to the government (local school boards, state 
legislature, or federal government). 

• Teachers and administrators also appear to be an important audience as 16 states 
direct their reports to teachers and administrators and other school personnel. 

• Only nine states specificahy mention parents as the target audience (though parents 
are probably included under those reports directed to the general public) 

• Reports directed at “other” audiences (n=6) included: special education community 
(from a special education document), students, researchers, and assessment 
personnel. 

Types of indicators used. Accountability reports use a wide range of indicators. 

We found a variety of common and unique educational indicators. Examples of common 
indicators included such things as detailed financial information (such as levy data and 
expenditures), staffing information (such as teacher education), and postsecondary outcomes (such 
as percentage of students attending college or employment rates). Some unique indicators included 
absenteeism rates, student mobility rates, and minutes spent in math and reading. 
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Meeting^State Standards. Some states report their assessment scores in terms of 
whether they met prescribed state standards (e.g., the percentage of students passing the 
standard of 50% on a nationally normed test). Our analysis of reports showed that 15 
states reported whether their students met the prescribed state standards: Alaska, 
Connecticut, Delaware, Georgia, Indiana, Kansas, Louisiana, Maryland, Michigan, 
Nebraska, New York, North Carolina, Ohio, Rhode Island, South Carolina. For the 
purposes of this analysis, we included those reports that mentioned standards, benchmarks 
or goals (including those in Goals 2000) when reporting data on students. 

Many states, however, are unclear as to the relationships among curriculum, 
standards, and assessrrients. Kansas makes the link very clear in its single accountability 
document, giving data on students’ performance as well as descriptions of innovative 
programs that demonstrate the link between curriculum and assessment. While it is clear 
in some of the reports that standards exist, it is often unclear whether students are meeting 
the standards, or what the specific standards are. 

What Do the Documents Look Like? 



In thoroughly examining the reports of the 48 states (including Washington DC) that sent 
documents to us, we found 48 different approaches - all with different indicators highlighted or 
excluded. Many states use tables, spreadsheets, and even the Internet to communicate their results. 
A uniform format does not exist. If states provide test data, then the methods of reporting those 
data also vary considerably. We did find that two sections usually are included in reports: some 
type of outcome data and financial data for regular education students. Examples of different 
formats can be found in Technical Report 20. Some formats are more understandable than 
others, and some formats may be more useful for one type of audience (e.g., researcher, school 
administrator) but not for others (e.g. parents, general public). 

Students with Disabilities in AccountabiUtv Reporting 

For every state that sent us a report, there was at least one educational indicator in at least 
one report for which it was not clear whether students with disabilities were included. As indicated 
in Table 1, of the 44 states that included input data, 13 included students with disabilities, either 
aggregated with the regular education data, or disaggregated as separate data. Similarly, for the 44 
states with process data, 30 included (aggregated or disaggregated) data on students with 
disabilities. Thus, the most commonly reported information for students with disabilities was 
process information. From the data collected, it is evident that the most common educational 
indicator used for special education students is enrollment (N=25). Enrollment data for special 
education students often includes detailed information on the number of students in each disability 
category, average daily membership, and referral data. Other special education indicators included 
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special programs, and detailed expenditures. A few states included such indicators as time spent in 
regular education versus separate settings, number of students exiting special education and 
reasons for exit, staff and case load ratios, and racial/ethnic gender special education classification 
rates. 

Outcome data, on the other hand, were aggregated or disaggregated for students with 
disabilities in only 11 of the 47 states plus Washington DC. In fact, close to half of the states 
provided information about the inputs/contexts and/or educational processes for students with 
disabilities, but did not mention anything about outcome data for students with disabilities. Table 2 
provides a summary of the kind of outcome data included by the 1 1 states that report on these data. 
As indicated, few of these states include students with disabilities in all of their outcome measures. 
For those that do disaggregate data, it is often not clear whether the total scores include students 
with disabilities. 



Ten states (Alabama, Arizona, Delaware, Georgia, Louisiana, Missouri, New Jersey, New 
York, Texas and West Virginia) specifically reported that they exclude students with disabilities’ 
outcome data from at least one of their reports or do not include data on students with disabilities 
on one outcome indicator. These states were very clear as to what their reporting practices are 
when it comes to students with disabilities. 

Accountability documents that reported on the exemption of students from testing were 
rare. Only eight states (Alaska, Arizona, Connecticut, Maryland, New York, Texas, Vermont, 
Washington) reported the number of students exempted from testing. Of those eight, four states 
specified the number of students with disabilities exempted from the test (Connecticut, New York, 
Texas, Washington). One state, Washington, requires districts to submit an “answer document” 
for every student that is enrolled. If a student is not tested, then districts must provide brief 
descriptive information about the student and indicate the reason he or she was not tested. Some of 
the reasons for exemption included: absent, special education student, limited English proficient 
student, withdrawn, student parent refusal, disruptive, temporary crisis, no reason given. 

Many states do not mention whether students with disabilities are included in state 
standards. However, five states report whether students with disabilities have met state standards 
(Connecticut, Delaware, Louisiana, New York, North Carolina). While the term “all students” is 
mentioned often, a definition of “all” usually is not given. 

Separate special education reports. Of the 47 states (plus Washington DC) that have a 
current accountability report, a handful devote a separate document to state special education 
student outcomes. Five states (Arkansas, Connecticut, Louisiana, New Jersey, and New York) 



[Insert Table 2 here] 




8 



9 



AERA/jm 



Educational Accountability Reporting 



report on various indicators of special education students. Only Connecticut, Louisiana, and New 
York have test-based outcome data in the separate special education report. Ohio has a separate 
special education report, but it does not provide state level data. Connecticut indicates that it 
creates its special education report because it has the “duty to provide opportunities for all students 
with disabilities to achieve statewide student goals” (Special Education in Connecticut 1994-95, CT 
State Dept, of Education, January 1996, p. 1). New Jersey reports that its sp>ecial education 
document is designed as a reference book to assist New Jersey’s special education community in 
planning and evaluating educational efforts for pupils with educational disabilities. New York and 
Connecticut also specifically state in their reports that they compile this information in order to meet 
state requirements to describe special education programs, and educational results for students with 
disabilities; however, only three states (Connecticut, Louisiana, New York) also provide test data 
for students with disabilities in their reports. Of special note, Connecticut and New York also 
report the number of students exempted from testing in their special education reports. Of the 
states that produce a special education report, two states (Connecticut and New York) include 
information on special education students in another accountability document that the state annually 
produces. 

Disaggregated special education student data . Of the 11 states that provide test-based 
outcome data on students with disabilities, 10 provide disaggregated test data. Eight states present 
these data in their regular accountability reports (Georgia, Kansas, North Carolina, New York, 
Rhode Island, South Carolina, Texas, and Virginia) Two states present their test-based outcome 
data only in a separate special education report (Connecticut and Louisiana). New York presents 
data in both types of reports. 



Discussion 

Variability is the best word to summarize what we found in our analysis of state reports. 
Accountability reports can contain a great deal of useful information, and serve many purposes for 
many different audiences. Many differences exist in the reporting of regular and special education. 
These differences are highlighted in Table 3. 

[Insert Table 3 here] 

Accountability systems will always vary by state, according to the needs and circumstances 
relevant to that state; however, even within this needed variability, there are some common 
recommendations that are relevant to all states: 

• Specify the target audience for each report, and gear the information to that 
audience. 

• If possible, gain input from the targeted audiences on information needs. 
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• Provide a “Pyramid of information” with a brief , easy-to-understand report for 
quick reference, while also making more detailed information available to those 
who need it. 

• Avoid a great deal of overlap between reports. 

• Make the purpose of the accountability system clear, and provide the reader with 
information on the appropriate and inappropriate use of information (such as 
whether to make comparisons between districts). 

• Choose educational indicators carefully and maintain some consistency among 
different reports. 

We also have the following suggestions for reporting on smdents with disabilities. 

• Be clear about the role of students with disabilities in data and standards. If making 
references to “all” smdents, then “all” smdents should include smdents with 
disabilities. 

• While special education reports can be very useful in providing detailed information 
about smdents with disabilities, general accountability reports should also include 
information on smdents with disabilities. 

• When making comparisons among schools, districts, and states, and even when 
making longimdind comparisons, comparisons should be based on similar 
populations. It is important to know the characteristics of the smdents whose 
scores are being compared, such as the number of smdents with disabilities, the 
socioeconomic stams of smdents, and other characteristics. 

Public Law 105-17, the reauthorization of IDEA, was recently passed by Congress and 
signed into law by President Clinton. It requires that smdents with disabilities be included in 
accountability reporting. Now is the time to look at how states approach accountability reporting, 
and incorporate suggestions for best practice in including data for smdents with disabilities in these 
reports. If smdents with disabihtifes are not included in accountability reporting, it is difficult for 
parents, educators, administrators, and public to know that these smdents are making adequate 
progress toward educational goals! Indeed, accountability reports will need to disaggregate the 
data of smdents with disabilities, choose educational indicators carefully, and provide clear 
information to all intended audiences. 
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Table 2: States that Report Test-Based Outcome Data for Students with Disabilities 


State 


Name of Test 


Notes on Reporting 


Connecticut 


Connecticut Mastery Test 


Statewide percentage of students and special education 
students at or above state goal. It is not clear whether 
students with disabilities were included in the general 
test scores. 




Connecticut Academic Performance 
Test 


Disaggregated data on students with disabilities. It is 
not clear whether students with disabilities were 
included in the general test scores. These data are in a 
separate special education report. 


Delaware 


Interim Assessment Program 


Aggregates all students with the exception of some 
Limited English Proficiency students and some special 
education students. Scores for students who were 
assessed with minor accommodations are aggregated 
into general test reporting and scores for students who 
were assessed with major accommodations were not 
included. Accountability report does not include 
definitions of major or minor accommodations. No 
disaggregated data are provided. 


Georgia 


Graduation Tests 


Includes aggregated and disaggregated data on students 
with disabilities. Excludes students with disabilities 
from comparisons to prior years. 






Students with disabilities are excluded from reporting 
on the Curriculum-Based Assessment Program. 






It is not clear whether students with disabilities are 
included in the Writing Assessment for grades 5 & 8, 
Advanced Placement Exams, or in standardized 
achievement testing. 


Kansas 


Kansas Assessments in 
Mathematics, Reading, & Writing. 


Lists the number of students with disabilities by 
category who were tested, but it is not clear whether 
these scores are aggregated into reported results. Does 
give disaggregated data for two outcomes. 


Louisiana 


CAT 5, ACT, SAT 


Reports aggregated data for students identified as 
gifted/talented, speech impaired, and/or 
hospital/homebound only. Other students with 
disabilities are excluded. 




Louisiana Educational Assessment 
Program (LEAP), & Louisiana 
High School Graduation Exit Exam. 


Disaggregates the percentage of students with 
disabilities attaining cutoff scores. It is not clear 
whether students with disabilities were included with 
the regular test scores. These data are in a separate 
special education report. 
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Table 2, cont. 


State 


Name of Test 


Notes on Reporting 


New York 


Pupil Evaluation Pro^am Tests in 
Reading, Writing and Math. 


Disaggregated data. It is not clear whether students 
with disabilities were included with the regular test 
scores. 




Regents Preliminary Competency 
Tests, Regents Competency Test 


Disaggregated data. It is not clear whether students 
with disabilities were included with the regular test 
scores. 




Occupational Education Proficiency 
Examinations in communication 
systems, production systems, 
transportation systems, and clothing 
and textiles. 


Reports data for students with disabilities and regular 
education students who are in vocational programs. 
Disaggregated data are presented in both separate and 
general reports. 


North Carolina 


End-of-grade and end-of -course tests 
in reading, writing and mathematics. 


Disaggregated data. It is not clear whether students 
with disabilities were included with the regular test 
scores. 






It is not clear whether students with disabilities were 
included in standardized testing or in the National 
Assessment of Educational Progress (NAEP). 


Rhode Island 


MATT 


Disaggregated and aggregated data. 


South Carolina 


MATT 


Disaggregated and aggregated data. 




South Carolina Basic Skills 
Assessment Program. 


Disaggregated and aggregated data 

It is not clear whether students with disabilities were 
included in reporting of Advanced Placement or 
college entrance examination (SAT/ACT) results. 


Texas 


Texas Assessment of Academic 
Skills (TAAS), TAAS/TASP 
Equivalency Test (secondary exit 
exam), End-of-course Biology 
Exam, ACT, SAT 


Disaggregated data. It is not clear whether students 
with disabilities were included with the regular test 
scores. 

Students may be exempted from the TAAS if they (1) 
have received a special education exemption as 
determined by a review committee and specified in the 
students’ lEP, or (2) have received a Limited English 
Proficiency exemption, as determined by a review 
committee. 


Virginia 


Virginia Literacy Passport Tests in 
reading, writing and mathematics for 
Grade 6. 


Disaggregated results. Students with disabilities who 
are pursuing a special diploma are not required to 
participate in the Literacy Testing Program. 






It is not clear whether students with disabilities are 
included in standardized testing or in Advanced 
Placement exams. 
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Table 3 Regular Versus Special Education in State Accountability Reporting 


Regular Education 


Students with Disabilities 


All states report outcome data. 


Few states report outcome data for students 
with disabihties. 


Most states report on a variety of 
educational indicators, with inputs, 
processes and outcomes included. 


Cost data (inputs) and enrollment (process) are 
the most commonly reported educational 
indicators. 


Most states report tegular and special 
education in the same report. 


Every state reports some data for which it is 
unclear whether students with disabilities were 
included. 




Of the five states with a special education 
report, two states do not mention students with 
disabilities in any other report. 
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